He Quit on Tuesday. By Wednesday He Was the Apocalypse.

Leave a CommentHe Quit on Tuesday. By Wednesday He Was the Apocalypse.Commentary

Jacob Coxon is twenty-seven, British, and as of last week the man who told America the computers are going to kill everyone. He resigned from Anthropic on September 8, posted that the labs are racing to self-improving superintelligence and gambling with our lives, then did the morning-show circuit.

Coxon says there’s a real, serious chance that AI could wipe out humanity. He insists he isn’t exaggerating and isn’t just trying to hype the technology. He thinks AI might start improving itself on its own as early as next year or 2028. He’s worried it could be used to attack power grids, or to help create biological weapons that could kill off the human race. He also made clear that the AI we have today can’t do these things, but he thinks the AI coming soon might.

That is his sermon, delivered as a guru. Here is his actual résumé.

The Résumé

Son of a professor of medieval German literature. Elite school. United Kingdom team at the International Mathematical Olympiad — silver in 2016, bronze in 2017. Cambridge mathematics, graduated around 2020, the year the big language models stopped being a parlor trick. London rationalist circles. A brief try at commodity trading. Then San Francisco.

He did not grow up inside the labs. OpenAI took him in 2023 through a six-month residency built for people who had not worked in the field. Math kid, pretraining — the unglamorous work of stuffing models with text, code, and pictures until they behave. Colleagues called him a normal researcher. Some accounts credit work around a major model release. He is not the man who signs the compute budget.

May of this year he jumped to Anthropic, the AI shop that markets itself as the responsible grown-up. Four months later he quit. He said he talked to bosses about safety, thought about moving to an alignment desk, then walked. Short tenure, so the Anthropic equity went back. He still has a slice from the first lab.

Qualified to do what?

Qualified to describe how pretraining works at two frontier companies. Qualified to repeat what people say in Slack when the cameras are off. A colleague who runs alignment testing publicly backed the scare: yes, they earnestly believe it could kill all humans; personal guess, more than 10 percent this decade. That is not Coxon inventing a cult. That is the cult agreeing with him.

Qualified to time the end of the world to 2027? No. Nobody is. Recursive self-improvement is a hypothesis with a marketing department. The jump from “the model writes code” to “the model invents a plague and turns off the lights” is a stack of ifs. He is honest that current systems are not extinction engines. The rest is a timeline he cannot prove and a policy he can: slow the race, maybe ban capability jumps, get Washington involved.

Will It Destroy the World?

Not this afternoon. Not because a twenty-seven-year-old with three years on the assembly line said so on television. The models are useful, expensive, and already good at cheating tests and writing malware drafts. That is a security problem and an industrial race. Peking is not pausing for a Cambridge moralist. A temporary ban on making the models smarter is a gift to the country that will not sign it.

What’s actually valuable about Coxon isn’t his dramatic warning that AI could destroy us. It’s what he revealed about how AI companies operate: the people building these systems privately talk about how dangerous they could be, and then go ahead and release bigger, more powerful versions anyway. Being scared of your product and launching it can happen at the same company, even in the same group chat.

That doesn’t prove the world is ending. What it proves is that the AI industry likes having it both ways: they enjoy the importance that comes from saying ‘this could be deadly,’ and they enjoy the huge company valuations that come from saying ‘this will transform the world.’ Coxon gave up a lot of money by leaving, and that deserves respect. But don’t take his predictions of disaster at face value.

Going forward

We have to pick one of two paths. We can treat AI as a vital national industry and stay ahead of other countries, or we can treat one person’s resignation letter as if it were sacred truth, slow down, and let a rival country pull ahead. (The rival doesn’t do ‘morning shows,’ meaning it doesn’t care about public hand-wringing or media warnings.) Coxon knows what it’s like inside his former workplace, so he’s worth listening to about that. But that doesn’t make him an authority on whether AI is about to bring on a catastrophe. We can’t know when or if disaster will come, and until it does, we’re still competing with other countries and should act like it.